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SYSTEMS AND METHODS FOR PROVIDiNG 
INTERACTIVE SPEAKER IDENTMCATION TRAINING 

RELATED APPLICATION 

[00011 This application claims priority under 35 U.S.C. § 1 19 based on U.S. Provisional 

Application No. 60/419,214, filed October 17, 2002, the disclosure of which is incorporated 

herein by reference. 

[00021 This application is related to U.S. Patent Application, Serial No. 10/ 

(Docket No. 02-4042), entitled "Continuous Learning for Speech Recognition Systems," filed 
concurrently herewith, and U.S. Patent Application, Serial No. 10/610,533 (Docket No. 02- 
4046), entitled "Systems and Methods for Improving Recognition Results via User- 
Augmentation of a Database," filed July 2, 2003, the disclosures of which are incorporated herein 
by reference. 

BACKGROUND OF THE INVENTION 

Field of the Invention 

[00031 The present invention relates generally to multimedia environments and, more 
particularly, to systems and methods for providing interactive speech identification training in a 
mtiltimedia enviroiunent. 
Description of Related Art 

[00041 Conventional speaker identification systems require a huge amount of training data to 
identify speakers based on audio data received firom the speakers. These systems typically train a 
set of models based on the training data. When audio data is received by one of these systems. 
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the system segments the audio data into speaker turns. The system then analyzes each of the 
speaker turns to determine whether it matches one of the models. If a speaker turn matches one 
of the models, the system labels the speaker tum with the name of the speaker. 
[0005] For proper speaker identification, the system requires a huge amount of training data. 
For each speaker, for example, the system requires several minutes of speech to generate a 
Gaussian mixture model that can later be used to locate segments of speech from the same 
speaker. In some situations, it is difficult to obtain a sufficient amount of training data to 
generate an accurate Gaussian mixture model. For example, it is difficult to obtain sufficient 
training data from speakers who do not routinely speak or who speak in only small bursts (i.e., 
one to two sentences at a time). 

[0006] As a result, there is a need for mechanisms to improve speaker identification results 
even for speakers where there is insufficient training data. 

SUMMARY OF THE INVENTION 
[0007] Systems and methods consistent with the present invention collect data for training a 
speaker identification system in an interactive manner. Users may be prompted to identify 
speaker turns for unidentified or incorrectly identified speakers. The data from the users may be 
used to generate new speaker models and/or update links to existing speaker models. The 
speaker models may then be used to correctly label speaker turns. 
[0008] In one aspect consistent with the principles of the invention, a system provides 
speaker identification training. The system generates speaker models and receives audio 
segments. The system identifies speakers corresponding to the audio segments based on the 
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speaker models. At least one of the audio segments has an unidentified or misidentified speaker 
(i.e., an audio segment whose speaker cannot be accurately identified). The system presents, to a 
user, audio segments that include an audio segment whose speaker is unidentified or 
misidentified and receives, from the user, the name of the unidentified or misidentified speaker. 
The system may use this information to subsequently identify the unidentified or misidentified 
speaker by name for future audio segments. 

[0009] In another aspect consistent with the principles of the invention, a speaker 
identification system is provided. The system includes an indexer and a server. The indexer is 
configured to generate speaker models, receive audio segments, and identify speakers 
corresponding to the audio segments based on the speaker models. The indexer is unable to 
correctly identify at least one of the speakers corresponding to the audio segments. The server is 
configured to receive, from a user, the name of an unidentified speaker of the speakers 
corresponding to the audio segments, and provide the name of the unidentified speaker to the 
indexer for identification of the unidentified speaker in subsequent audio segments. 
[00101 In a further aspect consistent with the principles of the invention, a computer-readable 
medium stores instructions executable by one or more processors for speaker identification 
training by a speaker identification system. The computer-readable medium includes instructions 
for generating speaker models based on training data; instructions for presenting, to a user, audio 
segments for which no speakers can be identified from the speaker models; instructions for 
obtaining, from the user, a name of a speaker for at least one of the audio segments; instructions 
for generating a new speaker model for the speaker based on the at least one audio segment; and 
instructions for associating the name of the speaker with the new speaker model. 
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[0011] In another aspect consistent with the principles of the invention, a speaker 
identification system includes an indexer, a database, and a server. The indexer receives speech 
segments, where each of the speech segments has a corresponding speaker. The indexer also 
creates documents by transcribing the speech segments and identifies the names of the speakers 
corresponding to the speech segments. The indexer is unable to correctly identify the names of at 
least one of the speakers corresponding to the speech segments. As a result, some of the speakers 
are identified speakers and others of the speakers are unidentified speakers. The database stores 
the documents. The server retrieves one or more of the documents from the database and 
presents the one or more of the documents to a user. The server receives, from the user, the 
name of one of the unidentified speakers and provides the name of the unidentified speaker to the 
indexer for subsequent identification of speech segments from the unidentified speaker. 



BRIEF DESCRIPTION OF THE DRAWINGS 
[0012] The accompanying drawings, which are incorporated in and constitute a part of this 
specification, illustrate the invention and, together with the description, explain the invention. In 
the drawings, 

[0013] Fig. 1 is a diagram of a system in which systems and methods consistent with the 
present invention may be implemented; 

[0014] Fig. 2 is an exemplary diagram of the indexer of Fig. 1 according to an 
implementation consistent with the principles of the invention; 

[0015] Fig. 3 is an exemplary diagram of a portion of the recognition system of Fig. 2 
according to an implementation consistent with the present invention; 
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[0016] Fig. 4 is an exemplary diagram of the memory system of Fig. 1 according to an 
implementation consistent with the principles of the invention; 

[0017] Figs. 5A-5C are flowcharts of exemplary processing for speaker identification 
training according to an implementation consistent with the principles of the invention; and 
[0018] Fig. 6 is a diagram of an exemplary graphical user interface by which a document may 
be presented to a user according to an implementation consistent with the principles of the 
invention. 

DETAILED DESCRIPTION 
[0019] The following detailed description of the invention refers to the accompanying 
drawings. The same reference numbers in different drawings may identify the same or similar 
elements. Also, the following detailed description does not limit the invention. Instead, the 
scope of the invention is defined by the appended claims and equivalents. 
[0020] Systems and methods consistent with the present invention permit users to aid in the 
training of a speaker identification system by, for example, correctly identifying unidentified and 
misidentified speakers. The systems and methods may use the user-identification of imidentified 
speakers to train new speaker models for improved recognition results. The systems and 
methods may use the user-identification of misidentified speakers to update links to existing 
speaker models. 

EXEMPLARY SYSTEM 
[0021] Fig. 1 is a diagram of an exemplary system 100 in which systems and methods 
consistent with the present invention may be implemented. System 100 may include multimedia 
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(MM) sources 1 10, indexer 120, memory system 130, and server 140 comiected to clients 150 via 
network 160. Network 160 may include any type of network, such as a local area network 
(LAN), a wide area network (WAN) (e.g., the Internet), a public telephone network (e.g., the 
Public Switched Telephone Network (PSTN)), a virtual private network (VPN), or a combination 
of networks. The various connections shown in Fig. 1 may be made via wired, wireless, and/or 
optical connections. 

[0022] Multimedia sources 110 may include one or more audio sources and/or one or more 
video sources. An audio source may include any source of audio data, such as radio, telephone, 
and conversations in any language. A video soiwce may include any source of video data with 
integrated audio data in any language, such as television, satellite, and a camcorder. The audio 
and/or video data may be provided to indexer 120 as a stream or file. 

[0023] Indexer 120 may include mechanisms for processing audio and/or video data. Indexer 
120 may include mechanisms that receive data from multimedia sources 1 10, process the data, 
perform feature extraction, and output analyzed, marked-up, and enhanced language metadata. In 
one implementation consistent with the principles of the invention, indexer 120 includes 
mechanisms, such as the ones described in John Makhoul et al., "Speech and Language 
Technologies for Audio Indexing and Retrieval," Proceedings of the IEEE, Vol. 88, No. 8, 
August 2000, pp. 1338-1353, which is incorporated herein by reference. 
[0024] Fig. 2 is an exemplary diagram of indexer 120 according to an implementation 
consistent with the principles of the invention. Indexer 120 may include training system 210, 
statistical model 220, and recognition system 230. Training system 210 may include logic that 
estimates parameters of statistical model 220 from a corpus of training data. The training data 
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may initially include human-produced data. For example, the training data might include one 
hundred hours of audio data that has been meticulously and accurately transcribed by a human. 
Training system 210 may use the training data to generate parameters for statistical model 220 
that recognition system 230 may later use to recognize future data that it receives (i.e., new audio 
that it has not heard before). 

[0025] The training data might also include audio data for which the speaker has been 
identified. Training system 210 may use the training data to generate parameters for statistical 
speaker models that recognition system 230 may later use to recognize speakers firom future data 
that it receives. To build a speaker model for a new speaker, training system 210 may use a 
speaker independent Gaussian mixture model with approximately 2,048 Gaussians. This 
Gaussian mixture model may be trained based on a lot of diversified speakers (e.g., speakers of 
different ages, different genders, etc.). Training system 210 may use a conventional expectation 
and maximization process to fit audio from a speaker to the model. Training system 210 may 
then use a conventional maximum a posteriori adaptation process to generate a final model for 
the speaker. 

[0026] Statistical model 220 may include acoustic models, language models, and speaker 
models. The acoustic models may describe the time-varying evolution of feature vectors for each 
sound or phoneme. The acoustic models may employ continuous hidden Markov models 
(HMMs) to model each of the phonemes in the various phonetic contexts. 
[0027] The language models may include n-gram language models, where the probability of 
each word is a function of the previous word (for a bi-gram language model) and the previous 
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two words (for a tri-gram language model). Typically, the higher the order of the language 
model, the higher the recognition accuracy at the cost of slower recognition speeds. 
[0028] The speaker models may include a pool of models that are used to identify speakers 
from their speech. Speech from a particular speaker may be compared to the speaker models to 
determine the likelihood that the speech was produced from each of the models. The model with 
the highest likelihood may be determined to correspond to that speech. The particular speaker 
corresponding to the model may be determined based, for example, on a mapping function that 
links speaker names to speaker models. 

[0029] Recognition system 230 may use statistical model 220 to process input audio data. 
Fig. 3 is an exemplary diagram of a portion of recognition system 230 according to an 
implementation consistent with the principles of the invention. Recognition system 230 may 
include speaker segmentation logic 310, speech recognition logic 320, speaker clustering logic 
330, and speaker identification logic 340. In one implementation consistent with the principles 
of the invention, the functions performed by speaker segmentation logic 310, speech recognition 
logic 320, speaker clustering logic 330, and speaker identification logic 340 are similar to the 
functions described in John Makhoul et al., "Speech and Language Technologies for Audio 
Indexing and Retrieval," Proceedings of the IEEE, Vol. 88, No. 8, August 2000, pp. 1338-1353, 
which was previously incorporated herein by reference. 

[0030] Generally, speaker segmentation logic 310 may distinguish speech from silence, 
noise, and other audio signals in input audio data. For example, speaker segmentation logic 310 
may analyze each thirty second window of the input data to determine whether it contains 
speech. Speaker segmentation logic 310 may also identify boundaries between speakers in the 
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input stream. Speaker segmentation logic 310 may group speech segments from the same 
speaker and send the segments to speech recognition logic 320. 
[0031] Speech recognition logic 320 may perform continuous speech recognition to 
recognize the words spoken in the segments that it receives from speaker segmentation logic 310. 
Speech recognition logic 320 may generate a transcription of the speech using statistical model 
220. Speaker clustering logic 330 may identify all of the segments from the same speaker in a 
single document (i.e., a body of media that is contiguous in time (from beginning to end or from 
time A to time B)) and group them into speaker clusters (or speaker turns). Speaker clustering 
logic 330 may then assign each of the speaker tums a unique label. 

[0032] Speaker identification logic 340 may identify the speaker in each speaker turn by 
name or, when the name cannot be determined, by gender. Speaker identification logic 340 may 
use the speaker models within statistical model 220 to identify speakers from their speech. For 
example, speaker identification logic 340 may compare speech from a particular speaker within a 
speaker turn to the speaker models to determine the likelihood that the speech was produced 
from each of the models. Speaker identification logic 340 may identify the model with the 
highest likelihood (above some threshold) as the model corresponding to that speech. Speaker 
identification logic 340 may include a mapping function that links speaker names to the speaker 
models so that speaker identification logic 340 can identify the speaker by name once a model 
has been identified. 

[0033] Returning to Fig. 1, memory system 130 may store documents from indexer 120 and, 
possibly, documents from clients 150. Fig. 4 is an exemplary diagram of memory system 130 
according to an implementation consistent with the principles of the invention. Memory system 
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130 may include loader 410, trainer 420, one or more databases 430, and interface 440. Loader 
410 may include logic that receives documents from indexer 120 and stores them in database 
430. Trainer 420 may include logic that sends speaker identification information, such as audio 
data with identification of the corresponding speaker, in the form of training data to indexer 120. 
[0034] Database 430 may include a conventional database, such as a relational database, that 
stores documents from indexer 120. Database 430 may also store documents received from 
clients 150 via server 140. Interface 440 may include logic that interacts with server 140 to store 
documents in database 130, query or search database 130, and retrieve documents from database 
130. 

[0035] Returning to Fig. 1, server 140 may include a computer (e.g., a processor and 
memory) or another device that is capable of interacting with memory system 130 and clients 150 
via network 170. Server 140 may receive queries from clients 150 and use the queries to retrieve 
relevant documents from memory system 130. 

[0036] Clients 150 may include personal computers, laptops, personal digital assistants, or 
other types of devices that are capable of interacting with server 140 to retrieve documents from 
memory system 130. Clients 150 may present information to users via a graphical user interface, 
such as a web browser window. 

EXEMPLARY PROCESSING 
[0037] Systems and methods consistent with the present invention permit users to assist in 
speaker identification training to improve recognition results of system 100. For example, the 
user may supply the name of an unidentified or misidentified speaker that may be used to retrain 
indexer 120. 
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[0038] Figs. 5 A-5C are flowcharts of exemplary processing for speaker identification 
training according to an implementation consistent with the principles of the invention. 
Processing may begin with a user desiring to retrieve one or more documents from memory 
system 130. The user may use a conventional web browser of client 150 to access server 140 in a 
conventional manner. To obtain documents of interest, the user may generate a search query and 
send the query to server 140 via client 150. Server 140 may use the query to search memory 
system 130 and retrieve relevant documents. 

[0039] Server 140 may present the relevant documents to the user. For example, the user 
may be presented with a list of relevant documents. The documents may include any 
combination of audio documents and video documents. The user may select one or more 
documents on the list to view. When this happens, the user may be presented with a transcription 
corresponding to the audio data from the document. The user may also be presented with the 
audio and/or video data corresponding to the transcription. 

[0040] Fig, 6 is a diagram of an exemplary graphical user interface (GUI) 600 by which a 
document may be presented to a user according to an implementation consistent with the 
principles of the invention. In one implementation, GUI 600 is part of an interface of a standard 
Internet browser, such as Internet Explorer or Netscape Navigator, or any browser that follows 
World Wide Web Consortium (W3C) specifications for HTML. 

[0041] GUI 600 may include a speaker section 610 and a transcription section 620. Speaker 
section 610 may identify boundaries between speakers, the gender of a speaker, and the name of 
a speaker (when known). In this way, speaker segments are clustered together over the entire 
document to group together segments from the same speaker under the same label. In the 
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example of Fig. 6, one speaker, Elizabeth Vargas, has been identified by name. Other speakers 
have been identified by gender and number. For example, if "male 1 " speaks again within the 
same document, his speech will also be labeled "male 1." Transcription section 620 may include 
a transcription of the document. In the example of Fig. 6, the document corresponds to video 
data from a television broadcast of ABC*s World News Tonight. 

[0042] GUI 600 may also include playback button 630. The user may select playback button 
630 to hear the audio (or possibly see the video) corresponding to the transcription in 
transcription section 620. To facilitate this, the user may select a portion of the transcription in 
transcription section 620 by, for example, using a mouse to highlight the portion. The user may 
then select playback button 630 to hear the audio (or see the video) corresponding to the selected 
portion of the transcription. 

[0043] GUI 600 may further include a correction button 640. The user may select correction 
button 640 when the user desires to supply the name of an imidentified speaker or correct the 
name of a misidentified speaker within speaker section 610. Sometimes, a speaker may be 
identified only by gender in speaker section 610 because indexer 120 (Fig. 1) can identify the 
speaker's gender but cannot determine the speaker's name. This may occur when indexer 120 has 
not yet generated a speaker model corresponding to the speaker due, for example, to a lack of 
audio data for the speaker. Sometimes, a speaker may be misidentified (i.e., labeled with the 
incorrect speaker name) in speaker section 610. This may occur when indexer 120 incorrectly 
matches the audio data from the speaker to a speaker model of another speaker. If the user 
desires, the user may provide the name of an imidentified speaker or correct the name of a 
misidentified speaker by selecting correction button 640 and providing the correct information. 
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[0044] GUI 600 may receive the infonnation provided by the user and modify the document 
onscreen. This way, the user may determine whether the information was correctly provided. 
GUI 600 may also send the speaker identification information to server 140. 
[0045] Retuming to Fig. 5 A, the user may express a desire to provide the name of an 
unidentified speaker or correct the name of a misidentified speaker. In doing so, the user may 
identify a segment, such as a speaker tum, within transcription section 620 (Fig. 6) (act 505). 
The user may identify the speaker tum by selecting the corresponding segment using, for 
example, a mouse or by selecting one of the labels within speaker section 610. The user may 
then select correction button 640 to initiate the correction. 

[0046] Client 150 may determine whether the user has selected to identify the name of an 
unidentified speaker or correct the name of a misidentified speaker (act 510). Client 150 may 
make this determination based on the current speaker label provided in speaker section 610 for 
the segment identified by the user. For example, if the current speaker label includes a gender 
identification, then client 150 may determine that the user has selected to identify the name of an 
unidentified speaker. If, on the other hand, the current speaker label includes something other 
than a gender identification, then client 150 may detennine that the user has selected to correct 
the name of a misidentified speaker. 

[0047] If the user desires to provide the name of an unidentified speaker, the user may input 
the name in any conventional manner (act 515) (Fig. 5B). Client 150 may provide the name and, 
possibly, other identifying information, as speaker identification information, to server 140. The 
other identifying information may include, for example, an indication of the audio segment to 
which the name corresponds, the audio (or video) data corresponding to the segment, and/or 
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other information that may be useful for training indexer 120 to later identify audio data from this 
same speaker. 

[0048] Server 140 may determine whether additional audio data is necessary or desired (act 
520). For example, server 140 may determine whether the amount of audio data associated with 
the audio segment labeled by the user is less than approximately four minutes of audio. As 
explained above, the minimum amount of audio data needed to build a good Gaussian mixture 
model (with over 2,000 Gaussians) is approximately four minutes of audio. 
[0049] If the audio data associated with the audio segment is less than four minutes in length, 
then server 140 may locate additional audio segments that may be from the same speaker (act 
525). Server 140 may identify these additional audio segments based on the speaker clustering 
perforaied by indexer 120. Using the speaker clustering, more, similar audio segments may be 
identified in the same document and in other documents within memory system 130. 
[0050] For each audio segment located, sever 140 may confirm with the user that the audio 
segment is from the same speaker identified by the user (act 530). Server 140 may continue to 
obtain additional audio segments and confirm them with the user imtil it obtains around four 
minutes or more of audio data or the user expresses a desire to cease the confirmations. 
[0051] Once server 140 collects a sufficient amount of audio data or the user stops the 
confirmations, server 140 may provide the audio data, along with the name of the speaker, to 
indexer 120 as training data. Indexer 120 may use the audio data to build a speaker model for the 
new speaker (act 535). As described above, indexer 120 may use a speaker independent 
Gaussian mixture model with approximately 2,048 Gaussians. This Gaussian mixture model 
may be trained based on a lot of diversified speakers (e.g., speakers of different ages, different 
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genders, etc.). Indexer 120 may use a conventional expectation and maximization process to fit 
the audio data from the speaker to the model. Indexer 120 may then use a conventional 
maximum a posteriori adaptation process to generate a final model for the speaker. 
[0052] Indexer 120 may add the speaker model to the pool of speaker models within 
statistical model 220 (Fig. 2) (act 540). Thereafter, indexer 120 may correctly identify new audio 
data that it receives corresponding to the new speaker. 

[0053] If, instead of providing the name of an unidentified speaker, the user desires to 
correct the name of a misidentified speaker, the user may optionally input the correct name in any 
conventional manner (act 545) (Fig. 5C). Client 150 may provide the correct name and, possibly, 
other identifying information, as speaker identification information, to server 140. The other 
identifying information may include, for example, an indication of the segment (e.g., speaker 
turn) to which the name corresponds, the audio (or video) data corresponding to the segment, 
and/or other information that may be useful for indexer 120 to identify the speaker model 
corresponding to the speaker. 

[0054] Server 140 may determine whether confirmation of additional audio data is necessary 
or desired (act 550). For example, a certain amount of audio data may be needed to accurately 
identify the incorrectly labeled speaker model. In this case, server 140 may identify other audio 
segments that are close to the current segment (i.e., the speaker tum that the user identified as 
mislabeled) (act 555). Server 140 may use information relating to the speaker clustering 
performed by indexer 120 to identify these other potentially mislabeled audio segments. For 
each segment located, sever 140 may confirm with the user that the speaker tum is also labeled 
incorrectly (act 560). Server 140 may continue to obtain additional audio segments and confirm 
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them with the user xintil it until some threshold is met or the user expresses a desire to cease the 
confirmations. 

[0055] Server 140 may then provide the audio data, along with the correct name of the 
speaker, to indexer 120. Indexer 120 may use the audio data to identify the speaker model from 
which the audio data was produced. Indexer 120 may then update the name associated with the 
identified speaker model (act 565). For example, a mapping may exist between the names of 
speakers and the corresponding speaker models in statistical model 220. Indexer 120 may update 
the mapping to identify the correct name of the speaker for that model. 
[0056] In an alternate implementation, the user knows only that the speaker is misidentified, 
but does not know the true identity of the speaker. For example, the user may indicate simply 
that the name of the speaker associated with the speaker tum is incorrect. In this case, indexer 
120 may simply remove the mapping of the incorrect speaker name to the corresponding speaker 
model. 

[0057] When indexer 120 thereafter comes across audio data corresponding to a speaker that 
has been identified by the user (either as a result of an unidentified or misidentified speaker tum), 
indexer 120 compares the audio data to the models in the speaker model pool. Indexer 120 may 
determine the likelihood that the audio data was created by each of the speaker models. Indexer 
120 may identify the speaker model with the highest likelihood (above some threshold) as the 
speaker model corresponding to the audio data. Indexer 120 may then use its mapping of speaker 
models to speaker names to identify the speaker of that audio data by name. Indexer 120 may 
also update the documents in memory system 130 with the speaker name(s) identified by the 
user. 
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CONCLUSION 

[0058] Systems and methods consistent with the present invention provide an interactive 
speaker identification training system. Via these systems and methods, users are permitted to 
assist in the speaker identification training by, for example, correctly identifying unidentified and 
misidentified speakers. The systems and methods may use the user-identification of unidentified 
speakers to train new speaker models for improved recognition results. The systems and 
methods may use the user-identification of misidentified speakers to update links to existing 
speaker models. 

[0059] The foregoing description of preferred embodiments of the present invention provides 
illustration and description, but is not intended to be exhaustive or to limit the invention to the 
precise form disclosed. Modifications and variations are possible in light of the above teachings 
or may be acquired from practice of the invention. 

[0060] For example, an exemplary graphical user interface has been described with regard to 
Fig. 6 as containing certain features consistent with the principles of the invention. It is to be 
understood that a graphical user interface, consistent with the present invention, may include any 
or all of these features or different features to facilitate user assistance in speaker identification 
training. 

[0061] Further, a graphical user interface has been described as performing certain functions. 
It is to be understood that some, if not all, of these functions may be performed by client 150 or 
server 140. 

[0062] While series of acts have been described with regard to Figs. 5A-5C, the order of the 
acts may differ in other implementations consistent with the principles of the invention. 

17 



Docket No. 02-4021 



[0063] Further, certain portions of the invention have been described as "logic" that performs 
one or more functions. This logic may include hardware, such as an application specific 
integrated circuit or a field programmable gate array, software, or a combination of hardware and 
software. 

[0064] No element, act, or instruction used in the description of the present application 
should be construed as critical or essential to the invention unless explicitly described as such. 
Also, as used herein, the article "a" is intended to include one or more items. Where only one 
item is intended, the term "one" or similar language is used. The scope of the invention is 
defined by the claims and their equivalents. 
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